Accept "pipeline" to generate SearchService2 on sync processed Manifests#623
Accept "pipeline" to generate SearchService2 on sync processed Manifests#623JackLewis-digirati wants to merge 32 commits into
Conversation
…rolls back if there's a failure
What this PR doesImplements the text-services pipeline integration (issue #617). When a manifest is created or updated with a
Once the text service finishes, a background SQS handler (
Key design decisions
Only IManifestAugmentor is a new interface introduced to abstract operations that modify Manifests. In addition to this the ManifestS3Manager class was refactored as it was doing 2 jobs - S3 saving and merging with IIIF-CS NQ. Now that we have an additional source of Manifest content in text-service it felt appropriate to split this down, rather than bloat ManifestS3Manager. It also allowed for consistent handling of "original" Manifest. Original manifests are saved if pipeline is specified, this follows logic introduced in #576 |
Use AddDistinctById() helper as used elsewhere, updated it to return the number of items added, change base type to make more accommodating and add optional hook to alter item on add. Set label on AutoComplete and SearchService if not set. Rather than iterate and read contexts, add search2 context manually as it's a published constant
Avoids secondary DB call when it can be done in one
Avoids fat interface where both text builder and text search clients are implemented as one, BackgroundHandler will call both but API will only ever call 1. Also allows for separate control of Timeouts etc per client
This keeps all the logic for building payload in client, meaning the caller doesn't need to know how this is constructed. This should keep it simple for future BackgroundHandler consumer to implement
We have a known format and expectation for JobIds, this more accurately represents it compared to a string.
* Save the status provided in message if not completed, rather than defaulting to Failed. The assumption is it would always be Failed but this change means we reflect the actual status and could handle new ones * Add helper for constructing pipelines, following pattern used for other test entities. * Update tests to use TextJobId
Shares commonalities between different handlers * Deserialise message and log if fail * Helper to retry or dismiss if record not known * Set log context
This means that only valid pipelines are passed down, making calling logic simpler. Takes into account name and config.
* Added shorter timeout for builder, as per comment on ManifestWriteService.RegisterAndSubmitPipelineJobs() * Renamed CreateOrUpdateJob to UpsertJob to use consistent terminology
Rather than fetch pipeline and later fetch Manifest, do it on one
ManifestS3Manager had 2 different concerns - saving/deleting etc of manifest in S3 and also merging manifest with content from DLCS. Now that content can come from text-services it makes sense to split the DLCS logic from ManifestS3Manager (int DlcsManifestMerger). Callers can now augment Manifests separately from the ManifestS3Manager and only use that for persistence. This avoids bloat from S3Manager while still allowing some logic being shared (like copying of originals)
Currently 2 implementations, for DLCS and TextService. Keeps a consistent signature for updating manifests and cleanly captures logic for applying augmentations
6caedb9 to
1053b9d
Compare
Resolves #617
What does this change?
This PR enables pipelines in iiif-presentation as well as implementing a connection to the text services project.
Note
This PR now wraps database operations in a transaction so that failures to submit a manifest correctly rolls everything back. Additionally, as text services requires S3 to be available from the moment of submission, the S3 is saved before pipeline submission. This then needs to roll back by deleting the staging manifest from S3
Database Migration
Note
Details of migration. What does it change? Is it breaking/non-breaking?
pipeline_jobstable that tracks jobs being submitted to text-services, while being extendable to other pipelines in the futureConfiguration Changes
Note
This PR introduces configuration changes.
TextServices:BuilderApiUrinullTextServices:BuilderApiTimeoutSecondsTextServices:SearchApiUrinullTextServices:SearchApiTimeoutSecondsAWS:SQS:TextJobQueueNamenull